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Abstract 


We present and empirically evaluate an efficient algorithm that learns to aggre¬ 
gate the predictions of an ensemble of binary classifiers. The algorithm uses the 
structure of the ensemble predictions on unlabeled data to yield significant perfor¬ 
mance improvements. It does this without making assumptions on the structure or 
origin of the ensemble, without parameters, and as scalably as linear learning. We 
empirically demonstrate these performance gains with random forests. 

1 Introduction 

Ensemble-based learning is a very successful approach to learning classifiers, including well-known 
methods like boosting d, bagging d, and random forests 0. The power of these methods has 
been clearly demonstrated in open large-scale learning competitions such as the Netflix Prize ||4| 
and the ImageNet Challenge 0. In general, these methods train a large number of “base” classifiers 
and then combine them using a (possibly weighted) majority vote. By aggregating over classifiers, 
ensemble methods reduce the variance of the predictions, and sometimes also reduce the bias Q. 

The ensemble methods above rely solely on a labeled training set of data. In this paper we propose 
an ensemble method that uses a large unlabeled data set in addition to the labeled set. Our work is 
therefore at the intersection of semi-supervised learning Eia and ensemble learning. 

This paper is based on recent theoretical results of the authors M- Our main contributions here are 
to extend and apply those results with a new algorithm in the context of random forests 0 and to 
perform experiments in which we show that, when the number of labeled examples is small, our 
algorithm’s performance is at least that of random forests, and often significantly better. 

For the sake of completeness, we provide an intuitive introduction to the analysis given in 0. How 
can unlabeled data help in the context of ensemble learning? Consider a simple example with six 
equiprobable data points. The ensemble consists of six classifiers, partitioned into three “A” rules 
and three “B” rules. Suppose that the “A” rules each have error 1/3 and the “B” rules each have error 
1 /6. If given only this information, we might take the majority vote over the six rules, possibly 
giving lower weights to the “A” rules because they have higher errors. 

Suppose, however, that we are given the unlabeled information in Table[2 The columns of this table 
correspond to the six classifiers and the rows to the six unlabeled examples. Each entry corresponds 
to the prediction of the given classifier on the given example. As we see, the main difference between 
the “A” rules and the “B” rules is that any two “A” rules disagree with probability 1/3, whereas the 
“B” rules always agree. For this example, it can be seen (e.g. proved by contradiction) that the only 
possible true labeling of the unlabeled data that is consistent with Table [TJanc/ with the errors of the 
classifiers is that all the examples are labeled ’h-’. 

Consequently, we conclude that the majority vote over the “A” rules has zero error, performing 
significantly better than any of the base rules. In contrast, giving the “B” rules equal weight would 

*We assume that (bounds on) the errors are, with high probability, true on the actual distribution. Such 
bounds can be derived using large deviation bounds or bootstrap-type methods. 
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result in an a rule with error 1 /6. Crucially, our reasoning to this point has solely used the structure 
of the unlabeled examples along with the error rates in Table [T] to constrain our search for the true 
labeling. 



A classifiers 

B classifiers 

Xi 
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+ 
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X4 
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+ 

X5 

+ 

+ 

- 

+ 

+ 

+ 

Xq 

+ 

+ 

- 

- 

- 

- 

error 

1/3 

1/3 

1/3 

1/6 

1/6 

1/6 


Table 1: An example of the utility of unlabeled examples in ensemble learning 


By such reasoning alone, we have correctly predicted according to a weighted majority vote. This 
example provides some insight into the ways in which unlabeled data can be useful; 

• When combining classifiers, diversity is important. It can be better to combine less accurate 
rules that disagree with each other than to combine more accurate rules that tend to agree. 

• The bounds on the errors of the rules can be seen as a set of constraints on the true labeling. 
A complementary set of constraints is provided by the unlabeled examples. These sets of 
constraints can be combined to improve the accuracy of the ensemble classifier. 

The above setup was recently introduced and analyzed in 0. That paper characterizes the problem 
as a zero-sum game between a predictor and an adversary. It then describes the minimax solution of 
the game, which corresponds to an efficient algorithm for transductive learning. 

In this paper, we build on the worst-case framework of 0 to devise an efficient and practical semi- 
supervised aggregation algorithm for random forests. To achieve this, we extend the framework to 
handle specialists - classifiers which only venture to predict on a subset of the data, and abstain 
from predicting on the rest. Specialists can be very useful in targeting regions of the data on which 
to precisely suggest a prediction. 

The high-level idea of our algorithm is to artificially generate new specialists from the ensemble. 
We incorporate these, and the targeted information they carry, into the worst-case framework of 0. 
The resulting aggregated predictor inherits the advantages of the original framework; 

(A) Efficient: Learning reduces to solving a scalable p-dimensional convex optimization, and 
test-time prediction is as efficient and parallelizable as p-dimensional linear prediction. 

(B) Versatile/robust: No assumptions about the structure or origin of the predictions or labels. 

(C) No introduced parameters: The aggregation method is completely data-dependent. 

(D) Safe: Accuracy guaranteed to be at least that of the best classifier in the ensemble. 

We develop these ideas in the rest of this paper, reviewing the core worst-case setting of 0 in Section 
and specifying how to incorporate specialists and the resulting learning algorithm in Section]^ 

Then we perform an exploratory evaluation of the framework on data in Section Though the 
framework of 0 and our extensions can be applied to any ensemble of arbitrary origin, in this 
paper we focus on random forests, which have been repeatedly demonstrated to have state-of-the- 
art, robust classification performance in a wide variety of situations Qo). We use a random forest 
as a base ensemble whose predictions we aggregate. But unlike conventional random forests, we 
do not simply take a majority vote over tree predictions, instead using a unlabeled-data-dependent 
aggregation strategy dictated by the worst-case framework we employ. 


2 Preliminaries 

A few definitions are required to discuss these issues concretely, following 0. Write [a]+ = 
max(0, a) and [n] = {1,2,..., n}. All vector inequalities are componentwise. 
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We first consider an ensemble T-L = {/ii,..., hp] and unlabeled data xi, ..., on which we wish 
to predict. As in 0 , the predictions and labels are allowed to be randomized, represented by values 
in [—1,1] instead of just the two values {—1,1}. The ensemble’s predictions on the unlabeled data 
are denoted by F; 


F = 


//ii(xi) 

\hp{xi) 


hi{x2) 

hp{x2) 





( 1 ) 


We use vector notation for the rows and columns of F; h, = {hi{xi), • • • , and Xj = 

{hi{xj), • • • , hp{xj))^. The true labels on the test data T are represented by z = (zi;...; z„) S 
[—1,1]". The labels z are hidden from the predictor, but we assume the predictor has knowledge of 
a correlation vector b € (0,1]^ such that T hi{xj)zj > bi, i.e. ^Fz > b. These p constraints 
on z exactly represent upper bounds on individual classifier error rates, which can be estimated from 
the training set w.h.p. when all the data are drawn i.i.d., in a standard way also used by empirical 
risk minimization (ERM) methods that simply predict with the minimum-error classifier 0 . 


2.1 The Transductive Binary Classification Game 

The idea of 0 is to formulate the ensemble aggregation problem as a two-player zero-sum game 
between a predictor and an adversary. In this game, the predictor is the first player, who plays 
g = (pi; P 2 ; ■ • •; gn), a randomized label gi G [— 1 , 1 ] for each example The adversary 

then sets the labels z G [—1, Ij" under the ensemble classifier error constraints defined by b. HThe 
predictor’s goal is to minimize the worst-case expected classification error on the test data ^.r.t. 
the randomized labelings z and g), which is just | (l — ^z^g). This is equivalently viewed as 
maximizing worst-case correlation ^z^g. To summarize concretely, we study the following game: 

V := max min — z^g (2) 

gG[-i.i]" ze[-i,i]", n 

iFz>b 

The minimax theorem ( 0 , p.l44) applies to the game and there is an optimal strategy g* such 

that min —z^g* > V, guaranteeing worst-case prediction error |(1 — F) on the n unlabeled 

zG[-i,i]", n ^ 

iFz>b 

data. This optimal strategy g* is a simple function of a particular weighting over the p hypotheses - 
a nonnegative p-vector. 

Definition 1 (Slack Function). Let a > (F be a weight vector over l-L (not necessarily a distribution). 
The vector of ensemble predictions is F^cr = (x^J^ o', ■■■, whose elements’ magnitudes are 

the margins. The prediction slack function is 


7 (cr, b) := 7 (cr) := -b^cr -b - ^ [|xj o\-l] (3) 

and this is convex in a. The optimal weight vector a* is any minimizer a* G argmin [ 7 ( 17 )]. 

a->0P 


The main result of 0 uses these to describe the minimax equilibrium of the game 0. 

Theorem 2 (0). The minimax value of the game 01 is V = — 7 ((T*). The minimax optimal 
predictions are defined as follows: for all j G [n]. 


9j ■= 9ji^ ) = 



xjcr* < 1 

\sgn(xjcr*) 

otherwise 


^Since b is calculated from the training set and deviation bounds, we assume the problem feasible w.h.p. 
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2.2 Interpretation 


Theorem [^suggests a statistical learning algorithm for aggregating the p ensemble classifiers’ pre¬ 
dictions: estimate b from the training (labeled) set, optimize the convex slack function 7 ( 17 ) to find 
cr*, and finally predict with gj{a*) on each example j in the test set. The resulting predictions are 
guaranteed to have low error, as measured by V. In particular, it is easy to prove 13 that V is at least 
maxi bi, the performance of the best classifier. 

The slack function ([^ merits further scrutiny. Its first term depends only on the labeled data and 
not the unlabeled set, while the second term ^ ~ ^]+ incorporates only unlabeled 

information. These two terms trade off smoothly - as the problem setting becomes fully supervised 
and unlabeled information is absent, the first term dominates, and cr* tends to put all its weight on 
the best single classifier like ERM. 


Indeed, this viewpoint suggests a (loose) interpretation of the second term as an unsupervised regu- 
larizer for the otherwise fully supervised optimization of the “average” error b^cr. It turns out that 
a change in the regularization factor corresponds to different constraints on the true labels z: 


Theorem 3 (13)- Let I 4 


max 

ge[-i.i]" 


mm 

zG [—Q;,a]” 

^Fz>b 


1 T f 

-z g for 
n 


any a > 0. 


mino.>op 


-b' 


ETi [l 





Then Va = 


So the regularized optimization assumes each Zi G [—cr, a]. For a < 1, this is equivalent to assum¬ 
ing the usual binary labels (a = 1), and then adding uniform random label noise: flipping the label 
w.p. i (1 — a) on each of the n examples independently. This encourages “clipping” of the ensemble 
predictions xja* to the cr*-weighted majority vote predictions, as specified by g*. 


2.3 Advantages and Disadvantages 

This formulation has several significant merits that would seem to recommend its use in practical 
situations. It is very efficient - once b is estimated (a scalable task, given the labeled set), the 
slack function 7 is effectively an average over convex functions of i.i.d. unlabeled examples, and 
consequently is amenable to standard convex optimization techniques 13 like stochastic gradient 
descent (SGD) and variants. These only operate in p dimensions, independent of n (which is ^ p). 
The slack function is Lipschitz and well-behaved, resulting in stable approximate learning. 

Moreover, test-time prediction is extremely efficient, because it only requires the p-dimensional 
weighting cr* and can be computed example-by-example on the test set using only a dot product 
in The form of g* and its dependence on u* facilitates interpretation as well, as it resembles 
familiar objects: sigmoid link functions for linear classifiers. 

Other advantages of this method also bear mention: it makes no assumptions on the structure of % 
or F, is provably robust against the worst case, and adds no input parameters that need tuning. These 
benefits are notable because they will be inherited by our extension of the framework in this paper. 

However, this algorithm’s practical performance can still be mediocre on real data, which is often 
easier to predict than an adversarial setup would have us believe. As a result, we seek to add more 
information in the form of constraints on the adversary, to narrow the gap between it and reality. 


3 Learning with Specialists 


To address this issue, we examine a generalized scenario in which each classifier in the ensemble 
can abstain on any subset of the examples instead of predicting ±1. It is a specialist that predicts 
only over a subset of the input, and we think of its abstain/participate decision being randomized in 
the same way as the randomized label on each example. In this section, we extend the framework of 
Section 2.1 to arbitrary specialists, and discuss the semi-supervised learning algorithm that results. 


In our formulation, suppose that for a classifier i G \p\ and an example x, the classifier decides 
to abstain with probability 1 — Vi{x). But if the decision is to participate, the classifier predicts 


4 





hi{x) G [—1,1] as previously. Our only assumption on {vi(xi),..., Vi(xn)} is the reasonable one 
that Vi(xj) > 0 , so classifier i is not a worthless specialist that abstains everywhere. 

The constraint on classifier i is now not on its correlation with z on the entire test set, but on the 
average correlation with z restricted to occasions on which it participates. So for some [ 65 ]^ G [0,1], 


^ / Vi{Xj) \ 




(4) 


Define Pi{xj) := distribution over j G [n]) for convenience. Now redefine our 

unlabeled data matrix as follows: 

( Pl{xi)hi{xi) Pl{x 2 )hi{x 2 ) ■■■ Pl{Xn)hl{Xn)\ 

\ ; ■■■ ] (5) 

Pp{xi)hp{xi) Pp{x2)hp{x2) ■■■ Pp{Xn)hp{Xn) J 

Then the constraints 0 can be written as ^Sz > bg, analogous to the initial prediction game 0 . 
To summarize, our specialist ensemble aggregation game is stated as 

Vs := min max —z^g ( 6 ) 

zG[-l,l]", gG[-l.l]" n 


iSz>bs 


We can immediately solve this game from Thm. with (S, bg) simply taking the place of (F, b). 
Theorem 4 (Solution of the Specialist Aggregation Game). The awake ensemble prediction w.r.t. 

p 

weighting a > 0^ on example Xi is [S^cr] ^ = n pj {xi)hj {xi)aj . The slack function is now 


ls{cr) 



(7) 


The minimax value of this game is Vs = maxCT>op [— 7 s(o’)] = —^s{'X*g). The minimax optimal 
predictions are defined as follows: for all i G [n]. 


[5S] 


I 


9s{(Xs) = 




\sgii([ST»j],) 

otherwise 


In the no-specialists case, the vector pi is the uniform distribution (4,..., 4) for any i G [p], and 
the problem reduces to the prediction game (j^. As in the original prediction game, the minimax 
equilibrium depends on the data only through the ensemble predictions, but these are now of a 
different form. Each example is now weighted proportionally to Pj{xi). So on any given example 
Xi, only hypotheses which participate on it will be counted; and those that specialize the most 
narrowly, and participate on the fewest other examples, will have more influence on the eventual 
prediction pi, ceteris paribus. 


3.1 Creating Specialists for an Algorithm 

We can now present the main ensemble aggregation method of this paper, which creates spe¬ 
cialists from the ensemble, adding them as additional constraints (rows of S). The algorithm, 
HedgeClipper, is given in Fig. [T] and instantiates our specialist learning framework with a ran¬ 
dom forest lO. As an initial exploration of the framework here, random forests are an appropriate 
base ensemble because they are known to exhibit state-of-the-art performance JTOl . Their well- 
known advantages also include scalability, robustness (to corrupt data and parameter choices), and 
interpretability; each of these benefits is shared by our aggregation algorithm, which consequently 
inherits them all. 

Furthermore, decision trees are a natural fit as the ensemble classifiers because they are inherently 
hierarchical. Intuitively (and indeed formally too El), they act like nearest-neighbor (NN) pre¬ 
dictors w.r.t. a distance that is “adaptive” to the data. So each tree in a random forest represents a 


5 






somewhat different, nonparametric partition of the data space into regions in which one of the labels 
±1 dominates. Each such region corresponds exactly to a leaf of the tree. 

The idea of HedgeClipper is simply to consider each leaf in the forest as a specialist, which 
predicts only on the data falling into it. By the NN intuition above, these specialists can be viewed 
as predicting on data that is near them, where the supervised training of the tree attempts to define 
the purest possible partitioning of the space. A pure partitioning results in many specialists with 
« 1 , each of which contributes to the awake ensemble prediction w.r.t. a* over its domain, to 
influence it towards the correct label (inasmuch as [ 65 ]^ is high). 

Though the idea is complex in concept for a large forest with many arbitrarily overlapping leaves 
from different trees, it fits the worst-case specialist framework of the previous sections. So the 
algorithm is still essentially linear learning with convex optimization, as we have described. 


Algorithm 1 HedgeClipper 

Input: Labeled set L, unlabeled set U 
1: Using L, grow trees T = {Ti, , Tp} 

(regularized; see Sec. |3.2| i 
2: Using L, estimate bs on T and its leaves 
3: Using U, (approximately) optimize 0 
to estimate 

Output: The estimated weighting crj, for 
use at test time 

Figure 1 : At left is algorithm HedgeClipper. At right is a schematic of how the forest structure is related 
to the unlabeled data matrix S, with a given example x highlighted. The two colors in the matrix represent ±1 
predictions, and white cells abstentions. 



3.2 Discussion 

Trees in random forests have thousands of leaves or more in practice. As we are advocating adding 
so many extra specialists to the ensemble for the optimization, it is natural to ask whether this erodes 
some of the advantages we have claimed earlier. 

Computationally, it does not. When Pj{xi) = 0, i.e. classifier j abstains deterministically on Xi, 
then the value of hj{xi) is irrelevant. So storing S in a sparse matrix format is natural in our setup, 
with the accompanying performance gain in computing S^cr while learning a* and predicting with 
it. This turns out to be crucial to efficiency - each tree induces a partitioning of the data, so the set 
of rows corresponding to any tree contains n nonzero entries in total. This is seen in Fig. 

Statistically, the situation is more complex. On one hand, there is no danger of overfltting in the 
traditional sense, regardless of how many specialists are added. Each additional specialist can only 
shrink the constraint set that the adversary must follow in the game 0. It only adds information 
about z, and therefore the performance Vs must improve, if the game is solved exactly. 

However, for learning we are only concerned with approximately optimizing 75 ( 0 ") and solving the 
game. This presents several statistical challenges. Standard optimization methods do not converge 
as well in high ambient dimension, even given the structure of our problem. In addition, random 
forests practically perform best when each tree is grown to overflt. In our case, on any sizable test 
set, small leaves would cause some entries of S to have large magnitude, ^ 1. This can foil an 
algorithm like HedgeClipper by causing it to vary wildly during the optimization, particularly 
since those leaves’ values are only roughly estimated. 

From an optimization perspective, some of these issues can be addressed by e.g. (pseudo)-second- 
order methods Ha , whose effect would be interesting to explore in future work. Our implementation 
opts for another approach - to grow trees constrained to have a nontrivial minimum weight per leaf. 
Of course, there are many other ways to handle this, including using the tree stracture beyond the 
leaves; we just aim to conduct an exploratory evaluation here, as several of these areas remain ripe 
for fumre research. 
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4 Experimental Evaluation 


We now turn to evaluating HedgeClipper on publicly available datasets. Our implementation uses 
minibatch SGD to optimize runs in Python on top of the popular open-source learning package 
scikit-learn, and runs out-of-core (n-independent memory), taking advantage of the scalabil¬ 
ity of our formulation. ^ The datasets are drawn from UCI/LibSVM as well as data mining sites 
like Kaggle, and no further preprocessing was done on the data. We refer to “Base RF” as the forest 
of constrained trees from which our implementation draws its specialists. We restrict the train¬ 
ing data available to the algorithm, using mostly supervised datasets because these far outnumber 
medium/large-scale public semi-supervised datasets. Unused labeled examples are combined with 
the test examples (and the extra unlabeled set, if any is provided) to form the set of unlabeled data 
used by the algorithm. Further information and discussion on the protocol is in the appendix. 

Class-imbalanced and noisy sets are included to demonstrate the aforementioned practical advan¬ 
tages of FIedgeClipper. Therefore, AUC is an appropriate measure of performance, and these 
results are in Table Results are averaged over 10 runs, each drawing a different random subsam¬ 
ple of labeled data. The best results according to a paired t-test are in bold. 

We find that the use of unlabeled data is sufficient to achieve improvements over even traditionally 
overfitted RFs in many cases. Notably, in most cases there is a significant benefit given by unlabeled 
data in our formulation, as compared to the base RF used. The boosting-type methods also perform 
fairly well, as we discuss in the next section. 



Figure 2: Class-conditional “awake ensemble prediction” (x^cr*) distributions, on SUSY. Rows (top to bot¬ 
tom): {IK, IQK, lOOA} labels. Columns (left to right): a = {1.0, 0.3, 3.0}, and the base RF. 

The awake ensemble prediction values x^cr on the unlabeled set are a natural way to visualize and 
explore the operation of the algorithm on the data, in an analogous way to the margin distribution in 
boosting One representative sample is in Fig. on SUSY, a dataset with many (5M) examples, 
roughly evenly split between ±1. These plots demonstrate that our algorithm produces much more 
peaked class-conditional ensemble prediction distributions than random forests, suggesting margin- 
based learning applications. Changing a alters the aggressiveness of the clipping, inducing a more 
or less peaked distribution. The other datasets without dramatic label imbalance show very similar 
qualitative behavior in these respects, and these plots help choose a in practice (see appendix). 

Toy datasets with extremely low dimension seem to exhibit little to no significant improvement 
from our method. We believe this is because the distinct feature splits found by the random forest 
are few in number, and it is the diversity in ensemble predictions that enables HedgeClipper to 
clip (weighted majority vote) dramatically and achieve its performance gains. 

On the other hand, given a large quantity of data, our algorithm is able to learn significant structure, 
the minimax structure appears appreciably close to reality, as evinced by the results on large datasets. 

5 Related and Future Work 

This paper’s framework and algorithms are superficially reminiscent of boosting, another paradigm 
that uses voting behavior to aggregate an ensemble and has a game-theoretic intuition QllIIl. There 
is some work on semi-supervised versions of boosting m , but it departs from this principled struc¬ 
ture and has little in common with our approach. Classical boosting algorithms like AdaBoost lIlTll 
make no attempt to use unlabeled data. It is an interesting open problem to incorporate boosting 
ideas into our formulation, particularly since the two boosting-type methods acquit themselves well 


^It is possible to make this footprint independent of d as well by hashing features OH, not done here. 
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Dataset 

# 

training 

HedgeClipper 

Random 

Forest 

Base RF 

AdaBoost 

Trees 

MART 

d 

Logistic 

Regression 

kagg-prot 

10 

0.567 

0.509 

0.500 

0.520 

0.497 

0.510 

100 

0.714 

0.665 

0.656 

0.681 

0.666 

0.688 

ssl-text 

10 

0.586 

0.517 

0.512 

0.556 

0.553 

0.501 

100 

0.765 

0.551 

0.542 

0.596 

0.569 

0.552 

kagg-cred 

100 

0.558 

0.518 

0.501 

0.528 

0.542 

0.502 

IK 

0.602 

0.534 

0.510 

0.585 

0.565 

0.505 

lOK 

0.603 

0.563 

0.535 

0.586 

0.566 

0.510 

ala 

100 

0.779 

0.619 

0.525 

0.680 

0.682 

0.725 

IK 

0.808 

0.714 

0.655 

0.734 

0.722 

0.768 

wla 

100 

0.543 

0.510 

0.505 

0.502 

0.513 

0.509 

IK 

0.651 

0.592 

0.520 

0.695 

0.689 

0.671 

covtype 

100 

0.735 

0.703 

0.661 

0.709 

0.732 

0.515 

IK 

0.764 

0.761 

0.715 

0.730 

0.761 

0.524 

lOK 

0.809 

0.822 

0.785 

0.759 

0.788 

0.515 

ssl-secstr 

10 

0.572 

0.574 

0.535 

0.563 

0.557 

0.557 

100 

0.656 

0.645 

0.610 

0.643 

0.637 

0.629 

IK 

0.687 

0.682 

0.646 

0.690 

0.689 

0.683 

SUSY 

IK 

0.776 

0.769 

0.764 

0.747 

0.771 

0.720 

lOK 

0.785 

0.787 

0.784 

0.787 

0.789 

0.759 

lOOK 

0.799 

0.797 

0.797 

0.797 

0.796 

0.779 

epsilon 

IK 

0.651 

0.659 

0.645 

0.718 

0.726 

0.774 

webspam-uni 

IK 

0.936 

0.928 

0.920 

0.923 

0.928 

0.840 

lOK 

0.967 

0.970 

0.957 

0.945 

0.953 

0.901 


Table 2: Area under ROC curve for HedgeClipper vs. supervised ensemble algorithms. 


in our results, and can pack information parsimoniously into many fewer ensemble classifiers than 
random forests. 

There is a long-recognized connection between transductive and semi-supervised learning, and our 
method bridges these two settings. Popular variants on supervised learning such as the transductive 
SVM ifTSll and graph-based or nearest-neighbor algorithms, which dominate the semi-supervised 
literature lO, have shown promise largely in data-poor regimes because they face major scalability 
challenges. Our focus on ensemble aggregation instead allows us to keep a computationally inex¬ 
pensive linear formulation and avoid considering the underlying feature space of the data. Largely 
unsupervised ensemble methods have been explored especially in applications like crowdsourcing, 
in which the method of gave rise to a plethora of Bayesian methods under various conditional 
independence generative assumptions on F EOll . Using tree structure to construct new features has 
been applied successfully, though without guarantees ED. 

Learning with specialists has been studied in an adversarial online setting as in the work of Freund 
et al. II 22 II . Though that paper’s setting and focus is different from ours, the optimal algorithms it 
derives also depend on each specialist’s average error on the examples on which it is awake. 

Finally, we re-emphasize the generality of our formulation, which leaves many interesting questions 
to be explored. The specialists we form are not restricted to being trees; there are other ways of 
dividing the data like clustering methods. Indeed, the ensemble can be heterogeneous and even 
incorporate other semi-supervised methods. Our method is complementary to myriad classification 
algorithms, and we hope to stimulate inquiry into the many research avenues this opens. 
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A Additional Information on Experiments 


A.1 Datasets 


Information on the datasets used: 


Dataset 

Data sizes 

{m/)n 

Dim. d 

Comments 

kagg-prot 

3751 

1776 

Kaggle challenge 12311 

ssl-text 

1500 

11960 

12 I 

kagg-cred 

150000 

10 

Kaggle challenge 12411 

ala 

1605 / 30956 

123 

LibSVM 

wla 

2477 / 47272 

300 

LibSVM 

covtype 

581012 

54 

LibSVM 

ssl-secstr 

83679 (unla¬ 
beled: 1189472) 

315 

m 

SUSY 

5000000 

18 

UCI 

HIGGS 

11000000 

28 

UCI 

epsilon 

500000 

2000 

PASCAL Large Scale Learning Challenge 2008 

webspam-uni 

350000 

254 

LibSVM 


All data from the challenges (e.g. kagg-cred) lacked test labels, so the results reported are aver¬ 
aged over 10 random splits of the training data. 


A.2 Algorithms 

In all cases, the random forests were grown with default parameters for the feature and data splits 
(bootstrap data sample of input data size, and ~ Vd features considered per split), and 100 trees was 
standard. Varying these changes the induced diversity of the trees/partitions and may fundamentally 
affect the output of our algorithm, but exploration of such aspects is left to future work. All the 
comparator algorithms were also tTin with scikit-learn’s default parameters - in many cases like 
RFs, they are fairly insensitive to parameter choice. 

To overcome the statistical issues discussed in Sec. |3.2[ we found we needed to enforce some 
regularization on the tree used. We chose to impose a constraint on the minimum number of training 
examples in any leaf of the tree. This constraint was imposed as a parameter to grow the forest; 
thereafter, we could use all resulting leaves as specialists. To avoid any leaf specialist weights 
being too large but still collect as many leaf specialists as possible, we set the minimum number of 
examples per leaf to 10 with > IK labeled examples, and to 4 otherwise. 

We also tried an alternative way of avoiding small specialists: to simply grow an unregularized forest 
and then hlter out leaves, selecting only large enough leaves as specialists. This generally performed 
comparably or worse, consistent with the intuition that the diversity in unregularized tree predictions 
often manifests largely on small leaves. 

As pointed out in the paper, estimating b is an important step in an implementation. Accordingly, 
we used a bootstrap sample to do so; this performed comparably to holding out a validation set 
from a constant fraction of the labeled data. We observe throughout that it seems to matter far less 
how well the ensemble is trained, and more how well b is estimated; so we elected to only keep a 
modicum of labeled data for actually training the ensemble, and most for estimation. 

A notable issue we encountered is the setting of the “noise” rescaling factor a. We found 
HedgeClipper to be relatively insensitive to the precise choice of a, so it essentially sufficed 
for our experiments to try three choices: {0.3,1.0, 3.0}. The last is > 1.0, and therefore does not 
have an interpretation in terms of uniform label noise, but it is certainly a valid computational tool. 

Which of these three as to choose? Generally, we found that choosing a = 0.3 does not hurt 
performance, because our performance goals are often met best by separating the class-conditional 
peaks as much as possible. This is dangerous for more class-unbalanced datasets like kagg-cred, 
however, for which the default a = 1.0 works best. A useful heuristic which we used to choose a 
is simply to look at the class-conditional awake ensemble prediction distribution as plotted in Fig. 
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1 ^ the distribution can be roughly estimated and plotted on the fly, and we can quickly ascertain 
sensible choices of parameters like a. These choices appear to matter less at larger scale. 


11 


